{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning by grid-search\n",
    "\n",
    "In the previous notebook, we saw that hyperparameters can affect the\n",
    "generalization performance of a model. In this notebook, we show how to\n",
    "optimize hyperparameters using a grid-search approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our predictive model\n",
    "\n",
    "Let us reload the dataset as we did previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the column containing the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We drop from our data the target and the `\"education-num\"` column which\n",
    "duplicates the information from the `\"education\"` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the dataset is loaded, we split it into a training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a pipeline as seen in the first module, to handle both numerical and\n",
    "categorical features.\n",
    "\n",
    "The first step is to select all the categorical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use a tree-based model as a classifier (i.e.\n",
    "`HistGradientBoostingClassifier`). That means:\n",
    "\n",
    "* Numerical variables don't need scaling;\n",
    "* Categorical variables can be dealt with an `OrdinalEncoder` even if the\n",
    "  coding order is not meaningful;\n",
    "* For tree-based models, the `OrdinalEncoder` avoids having high-dimensional\n",
    "  representations.\n",
    "\n",
    "We now build our `OrdinalEncoder` by passing it the known categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(\n",
    "    handle_unknown=\"use_encoded_value\", unknown_value=-1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use `make_column_transformer` to select the categorical columns and apply\n",
    "the `OrdinalEncoder` to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (categorical_preprocessor, categorical_columns),\n",
    "    remainder=\"passthrough\",\n",
    "    # Silence a deprecation warning in scikit-learn v1.6 related to how the\n",
    "    # ColumnTransformer stores an attribute that we do not use in this notebook\n",
    "    force_int_remainder_cols=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we use a tree-based classifier (i.e. histogram gradient-boosting) to\n",
    "predict whether or not a person earns more than 50 k$ a year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline(\n",
    "    [\n",
    "        (\"preprocessor\", preprocessor),\n",
    "        (\n",
    "            \"classifier\",\n",
    "            HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4),\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning using a grid-search\n",
    "\n",
    "In the previous exercise (M3.01) we used two nested `for` loops (one for each\n",
    "hyperparameter) to test different combinations over a fixed grid of\n",
    "hyperparameter values. In each iteration of the loop, we used\n",
    "`cross_val_score` to compute the mean score (as averaged across\n",
    "cross-validation splits), and compared those mean scores to select the best\n",
    "combination. `GridSearchCV` is a scikit-learn class that implements a very\n",
    "similar logic with less repetitive code. The suffix `CV` refers to the\n",
    "cross-validation it runs internally (instead of the `cross_val_score` we\n",
    "\"hard\" coded).\n",
    "\n",
    "The `GridSearchCV` estimator takes a `param_grid` parameter which defines all\n",
    "hyperparameters and their associated values. The grid-search is in charge of\n",
    "creating all possible combinations and testing them.\n",
    "\n",
    "The number of combinations is equal to the product of the number of values to\n",
    "explore for each parameter. Thus, adding new parameters with their associated\n",
    "values to be explored rapidly becomes computationally expensive. Because of\n",
    "that, here we only explore the combination learning-rate and the maximum\n",
    "number of nodes for a total of 4 x 3 = 12 combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    \"classifier__learning_rate\": (0.01, 0.1, 1, 10),  # 4 possible values\n",
    "    \"classifier__max_leaf_nodes\": (3, 10, 30),  # 3 possible values\n",
    "}  # 12 unique combinations\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid, n_jobs=2, cv=2)\n",
    "model_grid_search.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the best combination of hyperparameters found by the grid\n",
    "search using the `best_params_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters is: {model_grid_search.best_params_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the grid-search is fitted, it can be used as any other estimator, i.e. it\n",
    "has `predict` and `score` methods. Internally, it uses the model with the\n",
    "best parameters found during `fit`.\n",
    "\n",
    "Let's get the predictions for the 5 first samples using the estimator with the\n",
    "best parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_grid_search.predict(data_test.iloc[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we check the accuracy of our model using the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model_grid_search.score(data_test, target_test)\n",
    "print(\n",
    "    f\"The test accuracy score of the grid-search pipeline is: {accuracy:.2f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy and the best parameters of the grid-search pipeline are similar\n",
    "to the ones we found in the previous exercise, where we searched the best\n",
    "parameters \"by hand\" through a double `for` loop.\n",
    "\n",
    "## The need for a validation set\n",
    "\n",
    "In the previous section, the selection of the best hyperparameters was done\n",
    "using the train set, coming from the initial train-test split. Then, we\n",
    "evaluated the generalization performance of our tuned model on the left out\n",
    "test set. This can be shown schematically as follows:\n",
    "\n",
    "![Cross-validation tuning\n",
    "diagram](../figures/cross_validation_train_test_diagram.png)\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p>This figure shows the particular case of <strong>K-fold</strong> cross-validation strategy\n",
    "using <tt class=\"docutils literal\">n_splits=5</tt> to further split the train set coming from a train-test\n",
    "split. For each cross-validation split, the procedure trains a model on all\n",
    "the red samples, evaluates the score of a given set of hyperparameters on the\n",
    "green samples. The best combination of hyperparameters <tt class=\"docutils literal\">best_params</tt> is selected\n",
    "based on those intermediate scores.</p>\n",
    "<p>Then a final model is refitted using <tt class=\"docutils literal\">best_params</tt> on the concatenation of the\n",
    "red and green samples and evaluated on the blue samples.</p>\n",
    "<p class=\"last\">The green samples are sometimes referred as the <strong>validation set</strong> to\n",
    "differentiate them from the final test set in blue.</p>\n",
    "</div>\n",
    "\n",
    "In addition, we can inspect all results which are stored in the attribute\n",
    "`cv_results_` of the grid-search. We filter some specific columns from these\n",
    "results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.DataFrame(model_grid_search.cv_results_).sort_values(\n",
    "    \"mean_test_score\", ascending=False\n",
    ")\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us focus on the most interesting columns and shorten the parameter names\n",
    "to remove the `\"param_classifier__\"` prefix for readability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the parameter names\n",
    "column_results = [f\"param_{name}\" for name in param_grid.keys()]\n",
    "column_results += [\"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n",
    "cv_results = cv_results[column_results]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shorten_param(param_name):\n",
    "    if \"__\" in param_name:\n",
    "        return param_name.rsplit(\"__\", 1)[1]\n",
    "    return param_name\n",
    "\n",
    "\n",
    "cv_results = cv_results.rename(shorten_param, axis=1)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that we are tuning only 2 parameters, we can visualize the results as a\n",
    "heatmap. To do so, we first need to reshape the `cv_results` into a dataframe\n",
    "where:\n",
    "\n",
    "- the rows correspond to the learning-rate values;\n",
    "- the columns correspond to the maximum number of leaf;\n",
    "- the content of the dataframe is the mean test scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pivoted_cv_results = cv_results.pivot_table(\n",
    "    values=\"mean_test_score\",\n",
    "    index=[\"learning_rate\"],\n",
    "    columns=[\"max_leaf_nodes\"],\n",
    ")\n",
    "\n",
    "pivoted_cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the data in the right format, we can create the heatmap as\n",
    "follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "ax = sns.heatmap(\n",
    "    pivoted_cv_results,\n",
    "    annot=True,\n",
    "    cmap=\"YlGnBu\",\n",
    "    vmin=0.7,\n",
    "    vmax=0.9,\n",
    "    cbar_kws={\"label\": \"mean test accuracy\"},\n",
    ")\n",
    "ax.invert_yaxis()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The heatmap above shows the mean test accuracy (i.e., the average over\n",
    "cross-validation splits) for each combination of hyperparameters, where darker\n",
    "colors indicate better performance. However, notice that using colors only\n",
    "allows us to visually compare the mean test score, but does not carry any\n",
    "information on the standard deviation over splits, making it difficult to say\n",
    "if different scores coming from different combinations lead to a significantly\n",
    "better model or not.\n",
    "\n",
    "The above tables highlights the following things:\n",
    "\n",
    "* for too high values of `learning_rate`, the generalization performance of\n",
    "  the model is degraded and adjusting the value of `max_leaf_nodes` cannot fix\n",
    "  that problem;\n",
    "* outside of this pathological region, we observe that the optimal choice of\n",
    "  `max_leaf_nodes` depends on the value of `learning_rate`;\n",
    "* in particular, we observe a \"diagonal\" of good models with an accuracy close\n",
    "  to the maximal of 0.87: when the value of `max_leaf_nodes` is increased, one\n",
    "  should decrease the value of `learning_rate` accordingly to preserve a good\n",
    "  accuracy.\n",
    "\n",
    "The precise meaning of those two parameters will be explained later.\n",
    "\n",
    "For now we note that, in general, **there is no unique optimal parameter\n",
    "setting**: 4 models out of the 12 parameter configurations reach the maximal\n",
    "accuracy (up to small random fluctuations caused by the sampling of the\n",
    "training set)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we have seen:\n",
    "\n",
    "* how to optimize the hyperparameters of a predictive model via a grid-search;\n",
    "* that searching for more than two hyperparameters is too costly;\n",
    "* that a grid-search does not necessarily find an optimal solution."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}